Competing topological and Kondo insulator phases on a honeycomb lattice 
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We investigate the competition between the spin-orbit interaction of itinerant electrons and their 
Kondo coupling with local moments densely distributed on the honeycomb lattice. We find that the 
model at half-filling displays a quantum phase transition between topological and Kondo insulators 
at a nonzero Kondo coupling. In the Kondo-screened case, tuning the electron concentration can 
lead to a new topological insulator phase. The results suggest that the heavy-fermion phase dia- 
gram contains a new regime with a competition among topological, Kondo-coherent and magnetic 
states, and that the regime may be especially relevant to Kondo lattice systems with 5d-conduction 
electrons. Finally, we discuss the implications of our results in the context of the recent experiments 
on SmBe implicating the surface states of a topological insulator, as well as the existing experiments 
on the phase transitions in SmBe under pressure and in CeNiSn under chemical pressure. 
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Systems containing both itinerant electrons and local 
moments continue to attract intensive interests in mod- 
ern condensed matter physics. The antiferromagnetic ex- 
change coupling between the two components gives rise to 
the Kondo singlet ground state. Historically, the Kondo 
effect in a single local-moment impurity provided the un- 
derstanding of the resistivity minimum in metals as well 
as the physics of dilute magnetic alloys and quantum 
nanostructuresi 2 -. In the concentrated case, considera- 
tion of the Kondo effect and its competition with mag- 
netically ordered ground states has been playing a cen- 
tral role in the understanding of the novel phases and 
quantum criticality of heavy fermion materials^. For the 
half-filled limit of the Kondo lattice system, the Kondo 
effect gives rise to the paramagnetic Kondo insulator (KI) 
state 2 -^. 

Recently, the quantum spin Hall insulator in two di- 
mensions (2D) and the topological insulator (TI) more 
generally have attracted extensive interest^. These in- 
sulators have a charge excitation gap in the bulk, but 
support gapless surface states protected by time-reversal 
symmetry (TRS). The surface states constitute a heli- 
cal liquid where the spin orientation is locked with the 
direction of electron momentum^^. Although they are 
robust against weak disorders that preserve TRS, the sur- 
face states may be influenced by magnetic impurities. For 
example, the conductance of ID edge helical liquid of a 
2D TI in the presence of a single magnetic impurity can 
exhibit a logarithmic behavior at high temperatures and 
goes to the unitarity limit at T = due to the forma- 
tion of a Kondo singlet 1 '' 11 . This is in contrast to the 
Kondo problem in conventional Luttinger liquids, where 
even very weak Coulomb interaction leads to vanishing 
conductance at zero temperature 12 . Generally speaking, 
the Kondo screening of magnetic impurities on the sur- 
face of TPs may not necessarily be complete due to the 



SU{2) breaking of the spin-orbit coupling(SOC)i 3 -, and 
the effective RKKY interaction between the local mo- 
ments can be mediated by the edge carries, leading to 
an in-plane noncollinear and helical order— ~—. For mag- 
netic impurities in TI's, previous studies have focused on 
the effect of surface impurities, i.e., magnetic impurities 
positioned on the surface of TI's, or coupled effectively 
to the surface states 1 ^. Whether and how the bulk mag- 
netic impurities influence the properties of TI's is largely 
an open problem. 

From the perspective of heavy-fermion physics, very 
interesting properties are emerging from materials which 
involve 5d electrons, such as the pyrochlore P^Ii^Or 19 . 
The significant SOC of the 5d electrons may give rise 
to topologically non-trivial physics for the 5d electrons 
alone, raising the intriguing question of the interplay be- 
tween topological and Kondo physics. The regime of 
transitions among the competing ground states repre- 
sents a setting in which the effects of strong interac- 
tions on TI's may become more tractable. Furthermore, 
Kondo insulators themselves may become topological as 
a result of the symmetry properties of the hybridization 
matrij*2£. 

Motivated by these recent developments, in this let- 
ter we study a dense set of magnetic impurities interact- 
ing with the spin-orbit coupled itinerant electrons on the 
honeycomb lattice as illustrated in Fig. [TJ Such a system 
is relevant for the graphene/magnetic moment interface 
and could be constructed through cold atoms in an op- 
tical lattice. The system could also be realized by grow- 
ing a 2D TI on an appropriate magnetic insulating sub- 
strate; the similar heterostructures involving TI Bi2Sc3 
thin films and superconducting layers have already been 
fabricated by the molecular beam epitaxy technique 21 . 
It may very well be built based on the existing 5d elec- 
tron based iridates on the honeycomb lattice, such as 



FIG. 1: Itinerant electrons moving on a honeycomb lattice 
while coupled vertically to the local spins on a parallel lattice. 



FIG. 2: The unit cell and primitive translation vectors. 



Na2irO;j22,. Finally, given that recent experiments in 
SrnBg have provided tentative evidence for the surface 
states of a topological insulato r 23 ' 24 , our results here on 
the transitions between topological insulator and Kondo 
coherent states lead to the intriguing question of what 
happens to such surface states when SmBg and related 
intermetallic systems are tuned by external or chemical 
pressure (see below). 

The model we consider, illustrated in Fig|TJ is specified 
by the Hamiltonian 

(ij)o- -Cij»cr<T' 

+ JA-J>i-S ? i , (1) 
i 

where c\ a annihilates an electron at site i with spin 
component a =t,|, s\ = C; cr (<T crcr //2)ci (T ', and S\ repre- 
sents the local moments with a being the Pauli matri- 
ces. The parameters t and X so are the nearest neighbor 
hopping energy and the next-nearest-neighbor intrinsic 
(Dresselhaus-type) SOC of the conduction electrons re- 
spectively, with Vij = ±1 depending on the direction 
of hopping between the next-nearest-neighbor sites. Fi- 
nally, 3k is the antiferromagnetic Kondo coupling be- 
tween the spins of conduction electrons and local impuri- 
ties. The model Eq.([T|) minimally interpolates the Kane- 
Mele Hamiltonian (Jk = 0)2^ and the standard Kondo 
lattice Hamiltonian (A so = 0). We note that recent stud- 
ies have focused on the effect of Hubbard U interaction 
of the conduction electrons^—. 

To proceed, we note that the model of Eq.fl]) is con- 
nected to the Anderson lattice Hamiltonian 

H = Hkm + H cd + H d , (2) 

where Hkm is the Kane-Melc Hamiltonian [the first two 

terms of Eq.(l)], H cd = Vj2ia( c L d i<? + h - c -) is tne hy- 
bridization between the itinerant electrons and localized 
d-electrons, and H d = E J2 i(7 d\ a <ko + U £\ n dit n diir is 
for the local electrons with Eq being the local energy level 
and U the on-site Coulomb repulsion of local electrons. 
The models described by Eqs.fT]) and ((5J) are equivalent 
provided that, in the absence of SOC, the d-electrons are 
in the Kondo regime (U is sufficient large and Eq is well 



below the Fermi energy (Ep) of the conduction band). 
In this regime, J K ~ V 2 [ Ef ~e + u-eI+e ^ 0ur calcu " 
lations will be carried out in Eq.©. As our focus is on 
the competition between the TI and KI at half filling, we 
shall mainly consider the paramagnetic states. 

In the momentum k-space, the conduction elec- 
tron part of Hamiltonian takes the form of Hkm — 
Ek„ Cl,M t A„ with Ct = (< to ,4,k„) aad 

where, a — +1 and —1 refers to spin up and spin down, 
At = 2A so [sin£;i — sinfc2 — sin(fei — kz)], £k = — i(l + 
e 1 + e~ lk2 ). We have included the chemical potential 
/i-term to control the electron filling. The subscripts a 
and b denote two sublattices of the honeycomb lattice as 
shown in Figf2] Each unit cell has two adjacent a, b sites, 
and the primitive vectors are ai and a2. 

For the local electrons, we consider the large-C/ limit 
and utilize the slave-boson method^. The local elec- 
trons are expressed as d\ a = // CT &i, with f^ a and bi a 
being respectively fermionic and bosonic operators sat- 
isfying the constraint b\bi + fLf' 1 ^ = 1- Introducing 

the basis k = (c+ >kfr > 4,k«r> /a,ko-. fb,ka) in the k -sP a ce, 
the mean- field Hamiltonian is expressed as Hmf = 

Ek^L^ka^k. With 

/ M ka rV-I \ , , 

H ^-{rV-I (E + A) • I ) ■ (4) 

Here, / is a 2 x 2 identity matrix, r = (b) is the con- 
densation density of the bosons, and A is the Lagrange 
multiplier introduced to implement the constraint. We 
will carry out our calculations for N = 2 (a = ±1); a 
large- AT generalization in the presence of SOC may also 
be considered—. The quasiparticle bands of the mean- 
field Hamiltonian Eq. Q are degenerate for the two spin 
components. For each spin component, the Hamiltonian 
can be diagonalized (even though the matrix is 4 x 4) 
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FIG. 3: (a)Large system with periodic boundary condition: 
The mean-field parameter r as a function of V for A so = 
0, 0.15, and 0.25. As a comparison, the red open square is for 
the corresponding single-ion Kondo problem for A so = 0.15. 
The inset shows the critical V c as a function of A so . All the 
coupling constants are in unit of t. (b) The site-dependence of 
r for the lattice with zig-zag edges for several V in the Kondo 
phase, the width N2 = 40, A so = 0.15. 



giving rise to the quasiparticle dispersion 
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with Gk± = Eq + A + n ± x/A 2 . + |e k | 2 - The parameters 
r and A are determined by the following equations 
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with Af being the total number of unit cells and Jf in- 
dicating the real part. In the following we shall mainly 
consider the half-filled case, corresponding to fi = 0. 

The formation of the quasiparticle bands, specified by 
Eq.©, requires the renormalized hybridization V* = 
rV 7^ 0. By contrast, if V* — 0, the spectra sepa- 
rate into the decoupled conduction bands and local level. 
Moreover, the band inversion takes place at V* = 0. 
While this feature is hidden and not important in ordi- 
nary Kondo lattice problems, it is crucial in the present 
problem because now the conduction bands are from the 
TI. As a consequence, the bulk gap of TI closes at the 
onset of V * , leading to a quantum phase transition to the 
KI. 

At zero temperature, r ( or V* ) is non-zero only if V 
is larger than a critical V c , as a result of the suppressed 
density of conduction electron states for a honeycomb 



lattice. Figj3ja) shows the numerical results for the In- 
dependence of r for several values of A so . The local level 
Eq is taken at the bottom of the conduction band. The 
critical V c ~ 1.3 for A so = 0, and increases almost linearly 
with A so , as seen in the inset of Figj3^a). When V < V c , 
r = 0, indicating the Kondo destruction, so the system 
remains in the TI phase with a bulk gap Ax = 6\/3A so . 
While for V > V c , r ^ 0, the Kondo screening emerges 
and the band inversion takes place immediately, so the 
system enters into the KI phase. For small r, the KI 
phase has a finite hybridization gap A^- ~ 2r 2 V 2 /3t. 
This is the direct band gap at the F-point where the 
contribution from the SOC vanishes. 

It is interesting to compare the results here for the 
Kondo-lattice problem with those for its counterpart of 
a single ion magnetic impurity imbedded in the bulk of 
the 2D TI. Using the same method, and for X so = 0.15 
as an example shown in FigU^a), we find V c ~ 2.07 for 
the single ion Kondo screening which is much larger than 
V c ~ 1.45 determined here. In the absence of SOC, the 
finite V c is due to the fact that the electron host is a 
pseudo gap system so that the single ion Kondo screen- 
ing needs a nonzero Kondo coupling comparable to the 
gap amplitude^r— . The enhancement of Kondo effect 
comparing to the single ion Kondo screening is similar to 
the Kondo lattice with ci-wave superconducting conduc- 
tion electrons^. 

We next investigate the surface states of the finite sys- 
tem with boundaries. We take a 2D ribbon by cutting 
two zizag edges with width N2, while the size along ai 
remains infinite. Then the boson mean-field r is de- 
pendent on the coordinate 712 and the sublattices, and 
can be denoted by r a (n2) and rt>(ri2) respectively. We 
have rb{ri2) — r a (N2 — n%) due to the inversion symme- 
try. FigJ3Jb) shows the site-dependence of r a and r& for 
N 2 = 40 and A so = 0.15. A general feature is that r de- 
creases rapidly from the edge to the bulk. This feature is 
attributed to the gapless edge states. r(ri2) is almost flat 
away from the edges (5 < n 2 < 35) as shown in FigJ3]Jb), 
indicating that the finite size effect is relatively small for 
= 40. 

Figure @] shows the energy spectra with four sets of 
parameters; here, \i — is imposed. Figs. |U[a) and (b) 
display the spectra of the conduction electrons in the ab- 
sence of Kondo singlet (V* = 0) and for X so — 0,0.03, 
respectively. The edge states with a single Dirac point at 
the Fermi energy in FigQJb) manifest the TI phase^^. 
In comparison, Figs. 0] (c) and (d) are the spectra for 
V > V c . The Kondo-singlet formation is clearly reflected 
in the hybridization gap at half-filling and the relatively 
narrow flat bands near the Fermi energy (near the tran- 
sition point the flatness is measured by t/V* ). 

Furthermore, we observe that in the KI phase, the nar- 
row bands can be separated from the continuum by in- 
creasing the SOC, leading to a bulk gap at the 1/4- or 
3/4- fillings (achieved by tuning the chemical potential 

(ON 

/i 7^ 0). This is the direct band gap between and 



E^ or between E^' and E^> at the points (2tt/3, 4tt/3) 
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FIG. 4: The energy spectra for a ribbon of width N2 = 40. 
(a)V = and \ so = 0; (b)V = and A so = 0.03; (c)V = 1.7 
and A so = 0; (d)V = 1.7 and \ so = 0.03; 

and (47r/3, 27r/3), respectively, with the gap magnitude 
Akt ~ 6v / 3r 2 l/ 2 A so /^ 2 for the large bulk system. More- 
over, for the finite system with boundaries, the edge 
states emerge again with a Dirac point within the bulk 
gap. This feature is robust for a range of the chemi- 
cal potential \x corresponding to the 1/4- or 3/4- filling. 
We have also calculated the Z2 topological bulk invari- 
ant S2D following the monodromy approach developed 
in Refj2£. The result confirms that "E.2D = 1 at half- 
filling and S2D = —1 at 1/4- or 3/4- filling. Therefore, 
the insulating phase at 1 /4- or 3/ 4-filling is topologically 
non-trivial, and its surface states, while having the spin 
direction locked by the momentum, contain the contri- 
butions from both conduction and local electrons. Hence 
in the case of FigQJd) we have a new TI phase with the 
Kondo-singlet formation and a surface heavy-fermion he- 
lical liquid. 

We now consider the TI-KI transition around V c . In 
the present analysis at the saddle-point level, the on- 
set of Kondo effect is continuous (Fig. [3fa)). Corre- 
spondingly, the KI gap sets in continuously. By con- 
trast, on the TI side there is simply a decoupling of 
the conduction-electron and local-moment components. 
However, we show that the quantum fluctuations beyond 
the saddle point reduce the bulk gap from the TI side, as 
is described in some detail in the Supplementary Mate- 
rials (SM). The situation is similar to the case of single- 
impurity pseudo-gapped Kondo problem, for which nu- 
merical renormalization group calculations, for instance, 
establish a well-defined second-order phase transition for 
the Kondo-destruction quantum critical point^. In the 
lattice case, the RKKY interaction between the local 
moments, which is mediated by Kondo coupling in our 
model, will also induce magnetic order. Taking into ac- 
count the magnetic order will leave the KI phase intact; 



as is standard, the Kondo screening present in the KI 
phase quenches the local moments and their ordering ten- 
dency. In the TI phase, we have explicitly shown (in the 
SM) that an antifcrromagnetic order, characterized by 
the order parameter M, will reduce the TI bulk gap to 

A T - 2(3V3A so - J K M)(1 - This TI gap 



remains non-zero for a finite range of M and V; in other 
words, the TI phase remains stable in the presence of an 
antiferromagnetic order for a range of V < V c . Our re- 
sults can be understood based on general arguments: in 
the presence of magnetic order, the Z 2 topological invari- 
ant is replaced by two spin-Chern numbers which remain 
unchanged when the time reversal symmetry is broken 
by the magnetic order— Meanwhile, the surface states 
remain gapless unless the bulk gap closes^. 

While a detailed transitions among KI and TI phases 
on the one hand, and antiferromagnetic order on the 
other is beyond the scope of the present work, our work 
does reveal that the heavy-fermion phase diagram con- 
tains a hitherto unexplored new regime with a compe- 
tition among topological, Kondo-coherent and magnetic 
states; such competition involves the physics of Kondo 
destruction and associated local quantum criticality= 
In other words, when the magnetic order and related 
dynamical effects are incorporated in our analysis, the 
TI-KI transition discussed here will represent a regime 
where topological effects strongly interplay with the on- 
set of magnetism and Kondo coherence. The simplifi- 
cation that proximity to quantum criticality brings may 
very well make the interaction effects on the TI phase 
and its associated surface states more tractable. 

We close by noting that we have treated the hy- 
bridization to be k- independent. When the spin- and 
k-dependences of the hybridization is incorporated, part 
of the KI phase may itself become topological, as empha- 
sized by Ref3. 

We now briefly discuss our work in the context of 
4 /-electron-based Kondo insulators. In SmBg, it is 
known that a sufficiently large external pressure col- 
lapses the Kondo coherence in SmBg and turns it into 
an antiferromagnetic metallic stated. Combined with 
the recent experimental evidence in SmBg for the edge 
states of a topological insulato r 23 i 24 , this is reminiscent 
of the transition among the topological and Kondo co- 
herent/magnetic states implicated by the present study. 
An intriguing question then arises, which deserves the 
study of future experiments: what happens to the can- 
didate chiral edge states when SmB 6 is placed under 
external pressure? Along a similar line, CeNiSn is an- 
other intermetallic system believed to be a Kondo insu- 
lator. In CeNiSn, (negative) chemical pressure achieved 
through Pd- or Pt- substitution for Ni is known to induce 
a transition out of its Kondo insulator stated—. It will 
therefore be informative to explore surface states in the 
Ce(Pti_ :c Ni :E )Sn and Ce(Pdi_ K Ni a; )Sn series. Finally, it 
is worth noting that CePtSn has the distinction in that 
it involves 5d electrons with a large SOC. 

To summarize, we have considered the effect of SOC 
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of the conduction electrons in a Kondo-lattice system. 
Our study offers the first qualitative understanding of 
the competition between topological and Kondo insula- 
tor ground states on a simple and yet generic model in 
two dimensions. While our analysis has so far been pri- 
marily confined to the paramagnetic cases, our results 
already suggest that the overall phase diagram of heavy- 
fermion systems includes a new regime with competition 
among topological, Kondo-coherent and magnetic states. 
This regime should be particularly prominent for heavy 
fcrmion systems whose conduction electron band is asso- 
ciated with the strongly spin-orbit-coupled 5c? electrons. 
As such, our work opens up a new regime of physical 



interest for compounds based on iridium, platinum and 
related 5d elements. 
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Supplementary Material — Competing topological and Kondo insulator phases on a honeycomb lattice 



by: Xiao-Yong Feng, Jianhui Dai, Chung-Hou Chung, Qimiao Si 



In this Supplementary Material, we first derive an effective action of the Kondo lattice model with SOC by taking 
into account the quantum fluctuations of the slave bosons. We show that while the quantum fluctuations do not 
affect the Kondo insulator (KI) phase, they do reduce the bulk gap of the topological insulator (TI) phase. We then 
consider the effect of the magnetic order of the local spins, which breaks the time-reversal symmetry. We show that 
the magnetic order does not change the nature of the TI phase. The notations here follow those introduced in the 
main text. 



A. Effect of quantum fluctuations beyond the saddle point 



We start from expressing the Hamiltonian introduced in the main text in the slave-boson representation, 



(ij)a ^CijSxro-' ia 

+ V Y,i4Afto + fl b ^) + (^0 + A) E flfu, + A £ b\h 



t J \ ( -M + o-Ak e k \( Co, 



k<7 

V 



(SI) 
(S2) 
(S3) 



+ E (<k- qCT & l q /«,k CT + /I ;k(7 6 a , qCa , k _ q<T ) + (E a + /l,k CT /a,ka + A E 6* , q 6a, q (S4) 



akqcr 



akcr 



In the above expression, a = a or b is the index for sublattices, A is introduced to implement the no double occupation 
of local electrons. In this formulism, the bosons can accommodate zero modes, which are denoted by &l !q= o = 

b a ,q=a = ' . Hence a nonvanishing r corresponds to the Bose-Einestcin condensation of the slave-bosons. Quantum 
fluctuations are contributed mainly from the bosons with non-zero q. The effective action S e f / for the system, defined 
by Tre-' 3H = J Vc^Vc^Vf^Vf^Vb^Vb^e-^ff, is then given by 



Seff = S c + ./lk»(-^ + E + \)fcM«>) + E & a,qH fi + X ^ 



Qq^0,Sl 



+ 



m (oj) + c.c 



Here, 



& = E^k^MXkaM) 



kcjo 



q/o,f2 



-iw — M + 



(S5) 
(S6) 



£k \ f C at k a (u) 

-iui - [i- crA k J \ c hjk(T (u;) 



(S7) 



is the bare action of the itinerant electrons, 



(2w+i)7r ^ q _ 2mr^ are M a tsubara frequencies for fcrmions 
(bosons). After integrating out the fermionic (Grassman) fields / Q . k(T (u;) and /* k(7 (w), the effective action becomes 

V 2 r 2 



s c+ E U-m+^-Y, _ iw+Eo+x 

, ^ -iuj + E a + A ' 



c a,ka( w ) c «,k<T(w) 



(S8) 



4, k - qCT (w - 0)c a , k _ qV (w - n0&* (n)& a , q ,(fi')- (S9) 
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We integrate out the bosonic fields and, to order of V 2 , obtain the following effective action for the conduction 
electrons 



y2 r 2 

-iuj + Eo + X 

O.K.UJI7 

V 2 



Ca,k CT (^)c«,k<r(^) 



( W )c a , kff ( W )+0(F 4 ). 



Now using the identities, 



(ifl - \)[i(uj + (1) - E — X] 
f3^iVL-X 



i(u> + 0) - E Q - X 



iui — Eq \ ifl — X 
= -n B (X) = 0, 

= n F (E a + X), 



i(ui + fi) - E - X 



we arrive at the following expression for the effective action 

Self = E«k.M,< kCT M)G ff (k,u;)- 1 



where the inverse of the Green's function is 

n F (E + \)V 2 , V 2 r 2 



C a ,ka(w) 



-10J + 



iuj — Eo 



IOJ — Eq 



- ji + crAj, 



n F (£ +A)V 2 



+ 



2(xJ — So — A 



ctAl 



(S10) 
(Sll) 

(S12) 
(S13) 

(S14) 
(S15) 
(S16) 



Since the bare energy level of the local /-electrons Eq < is much lower than the Fermi energy, the poles of the 
Green's function are essentially determined by the condition E + A <~ when r^O. In this case, one recovers all the 
mean- field results as discussed in the main text, where the quasiparticle bands are equivalently given by the poles of 
the Green's function. The Kondo regime where r ^ is determined by V > V c , with V c being self-consistently solved 

by the mean field equations. Quantum fluctuations contributed from the term nF ^°^^ V are negligible. Therefore 
we conclude that the existence of the KI phase and its properties are robust to the quantum fluctuations of the slave 
bosons. 

On the other hand, when V < V c , or r = 0, one has np(Eo + A) = i, then the poles of the Green's function are 
given by 



w 2 



uj 3 = 



"(A 4 ' 



-(A' + ^k + kkl 2 
-(^-v^k + kkl 2 





Ak + 


kkl 2 


+ £ ) 2 


+ 2V 2 


(a* - \ 


A k + 


kkl 2 


+ E ) 2 


+ 2V 2 


(p + \ 


A k + 


kkl 2 


+ E ) 2 


+ 2V 2 




A k + 


M 2 


+ E ) 2 


+ 2V 2 



(S17) 
(S18) 
(S19) 
(S20) 



At the half filling, fi = 0, there are two quasiparticle bands near the Fermi energy, uj\ and cj 2 - While the bands 
W3 and UJ4 are well below the Fermi energy and are always occupied. When V = 0, one simply recovers the two 
electron bands decoupled from the local spins, with a direct band gap between uj\ and w 2 opens at the Dirac points, 
12 — 3\/3A so . When V(< V c ) is small, the band structures do not change, while the amplitude 

v 2 
%■ 



\/AT 



e k 



of the band gap is given by A T (1 — j^?)- Therefore, the hybridization V(< V c ) just reduces the bulk gap of the 



TI phase. Combined with the result shown in Fig. 3(a), our results suggest that the bulk gap decreases continuously 
when the quantum critical point is approached from either sides. Therefore we argue that quantum fluctuations turn 
the TI-KI transition into a canonical continuous form, with vanishing gaps from both sides. 
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B. Effect of antiferromagnetic order of the local spins 



We turn consider an antiferromagnetic order of the local spins induced by the RKKY interactions. Our main 
concern here is how such an order, which breaks the time-reversal symmetry, influences the topological order. To 
address this issue, it is adequate to consider the effect of an effective staggered magnetic field associated with the 
antiferromagnetic order. For definiteness, we consider the antiferromagnetic order to correspond to the local spins 
staggered on the sublattices a and b. We use M to represent the magnetization of the /-electrons per site in the even 
(or odd) sublattice, which will induce a conduction-electron polarization by the Kondo interaction. This will add an 
additional term The effective Hamiltonian is: 

Hj = J2 ffI R M fLjaMa ~ aI * M fbMafbM ~ E ° J « Mc Im C ^ + E ffJ * Mc Ur C M«' ( S21 ) 
akcr fckcr akcr bkcr 

to the total Hamiltonian. Here, Ir is the RKKY interaction ~ PqJk , which is small for small hybridization. 

We follow the approach similar to what was described above, obtaining the inverse Green's function in the TI phase 
(V<V C ) 

V e k ~*w + \^ Eo+alR J M - <r(A k - JkM) ) 

Because in the absence of IrM : (Eq + A) is close to at half- filling, it is reasonable to replace uf{Eq + A + ctIrM) by 
1/2-crijjM when M is very small. Then, with the first order approximation, — ^^E^-IinM ~ ~ 2(iu>~E ) ~ q "i"^E ' 
Therefore, the poles of the Green's function is given by 

V 2 IrMV 2 .. ,,, ir V 2 IrMV 2 .. l2 

+ (Afc- J K M)][-u+-. -—+ H = -(A k -J K M)} = \e k \ 2 . 



1 2{u)-E ) uj-E y nl 2{uj-E ) uj-E q 

Obviously, the quasiparticle bands are similar to those of the original Tl phase. At the gap position, e k = 0, 

• 2 (lT2/ a Ai 
2(oj-Bo) 



w — — i^-prf^ = ^(Afe — JkM). The effect of the staggered fields (M) is evident: while M simply reduces 



the energy it also effectively renormalizes V 2 — > V 2 (l =F 2IrM). They both reduce the direct bulk gap as 

2(3\/3A so — JkM)[1 — V ^^ 2M - > ] at the Dirac points. Because the magnetic order breaks the time reversal symmetry 

(TRS), the spin degeneracy of the surface states splits^. It is well-known that the TI with a TRS-breaking magnetic 
field is characterized by the spin Chern number— ~— , and the Z2 characterization is recovered continuously when 
M — > 0^ 6 . Hence the TI phase persists when M 7^ unless the bulk gap A closes at certain large and finite 
hybridization {IrM and JkM). Therefore, we conclude that for small hybridization, the antiferromagnetic order 
keeps the KI phase intact. This demonstrates the validity of the main conclusions obtained from the SBMF method. 
For sufficient large hybridization {IrM and JkM), however, the system enters the antiferromagnetic phase in which 
the surface states may be gapped^. The band structures at that regime are rich but beyond the scope of the present 
paper. 
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